Head-mounted display by integration of phase-conjugate material

ABSTRACT

This invention has incorporated projective optics and phase conjugate material thus eliminating the requisite use of an external phase conjugate material to provide a see-through head mounted projective display. A key component of the invention is the use of optical imaging technology in combination with projective optics to make this revolutionary technology work. In previous head mounted projective displays the phase conjugate material had to be placed in the environment to display images, but in this invention one is not limited to the use of an external phase conjugate material but further extends its use to outdoor see-through augmented reality to produce images using the see-through head mounted projective display system. Furthermore, this invention extends the use of projective head mounted displays to clinical guided surgery, surgery medical, an outdoor augmented see-through virtual environment for military training and wearable computers.

This invention relates to a head mounted projection display (HMPD), andin particular to a compact lens assembly having a projection displayinterior of the HMPD for a teleportal augmented reality system and thisapplication is a continuation-in-part (CIP) of U.S. patent applicationSer. No. 10/090,070 filed Mar. 1, 2002 now U.S. Pat. No. 6,731,434 andwas funded in part by grant number 6502562 awarded by the Army STRICOMSNE.

BACKGROUND AND PRIOR ART

Networked virtual environments allow users at remote locations to use atelecommunication link to coordinate work and social interaction.Teleconferencing systems and virtual environments that use 3D computergraphic displays and digital video recording systems allow remote usersto interact with each other, to view virtual work objects such as text,engineering models, medical models, play environments and other forms ofdigital data, and to view each other's physical environment.

A number of teleconferencing technologies support collaborative virtualenvironments which allow interaction between individuals in local andremote sites. For example, video-teleconferencing systems use simplevideo screens and wide screen displays to allow interaction betweenindividuals in local and remote sites. However, wide screen displays aredisadvantageous because virtual 3D objects presented on the screen arenot blended into the environment of the room of the users. In such anenvironment, local users cannot have a virtual object between them. Thisproblem applies to representation of remote users as well. The locationof the remote participants cannot be anywhere in the room or the spacearound the user, but is restricted to the screen.

Head-mounted displays (HMDs) have been widely used for 3D visualizationtasks such as surgical planning, medical training, or engineeringdesign. The main issues of the conventional eyepiece-based HMDtechnology include tradeoffs between resolution and field-of-view (FOV),and between compactness and eye clearance, the presence of largedistortion for wide FOV designs, the conflict of accommodation andconvergence, the occlusion contradiction between virtual and realobjects, the challenge of highly precise registration, and often thebrightness conflict with bright background illumination. The concept ofhead-mounted projective displays (HMPDs) is an emerging technology thatcan be thought to lie on the boundary of conventional HMDs, andprojective displays such as the CAVE technology.

The basic HMPD concept of projection head-mounted display was earlydisclosed by Fisher Nov. 5, 1996, in U.S. Pat. No. 5,572,229.

Also a first international presentation was done by Kijima and Ojika in1997 [See Kijima and Ojika, “Transition between virtual environment andworkstation environment with projective head-mounted display.”Proceedings of IEEE 1997 Virtual Reality Annual International Symposium,IEEE Computer Soc. Press. 1997, pp. 130–7. Los Alamitos, Calif., USA.].

Also on Apr. 15, 1997, a U.S. Pat. No. 5,621,572 was also issued toFerguson on the conceptual idea of a display, i.e. optical, system forhead mounted display using phase conjugate material and method ofdisplaying an image.

Independently, the technology of HPMD was developed by Parsons andRolland as a tool for medical visualization [See Parsons and Rolland, “Anon-intrusive display technique for providing real-time data within asurgeon's critical area of interest. “Proceedings of Medicine MeetsVirtual Reality 98, 1998, pp. 246⁻²⁵¹”]. After the initial proof ofconcept using off-the-shelf components, a first-generationcustom-designed HMPD prototype was built to investigate perceptionissues and quantify some of the properties and behaviors of phaseconjugate materials in an imaging system. Since, the projection systemof the first-generation prototype was custom designed using adouble-Gauss lens structure and built from commercially availablecomponents. The total weight of each lens assembly was about 50 grams(already a significant reduction compared to using off-the-shelf optics)with mechanical dimensions of 35 mm in length by 43 mm in diameter.

Common to all these teleconferencing systems is the use of lenses ofvarious configurations and weights with distortions, lack of clarity andsmearing of the televised images. Representative of lenses that might atfirst glance appear to be useful in the teleconferencing systems arealso shown in:

U.S. Pat. No. 5,526,183 by Chen who teaches the use of a lens combiningdiffractive elements of both glass and plastic to reduce the weight andsize of the lens within a conventional helmet mounted display ratherthan the necessary projective helmet mounted display;

U.S. Pat. No. 5,173,272 by Aoki which discloses a four element highaperture lens with glass elements making it too heavy for helmetmounting;

U.S. Pat. No. 4,753,522 by Nishina et al which lens features all 4plastic elements and is fully symmetrical which latter property isimposed by its restricted application—a copy machine lens; and,

U.S. Pat. No. 4,669,810 by Wood which shows a head-mounted display withmany (more than 4) optical elements in the relay optics.

Consequently, there is a need for a HMPD augmented reality display thatmitigates the above mentioned disadvantages and has the capability todisplay virtual objects and environments, superimposes virtual objectson the “real world” scenes, provides “face-to-face” recording anddisplay, be used in various ambient lighting environments, and correctsfor optical distortion, while minimizing weight, computational power andtime. Lightweight, compactness, enhanced mobility and improved fidelityof the field of view are always of basic importance and/or highlydesirable, particularly, for head-mounted devices.

SUMMARY OF THE INVENTION

The first object of the present invention is to provide a HMPD withphase conjugate material integrated for use of see-through augmentedreality within the HMPD.

The second object of this invention is to allow extension of the HMPD tomobile outdoors environment, as well as those environments in which thephase conjugate material can not be used in the environment, such assurgical procedures.

The third object of this invention is to provide a user of the HMPD themeans of a mobile teleportal augmented reality system with or withoutthe use of phase conjugate material located in the environment.

A preferred embodiment of the invention encompasses a head mountedprojection display (HMPD) comprising in combination: a componentassembly for displaying computer generated image from a micro display;an optical assembly for projecting virtual images and said computergenerated images to a user's eye or eyes for monocular or binocularviewing; phase conjugate material for receiving and projecting saidvirtual images; an imaging lens for magnification of said phaseconjugate material; and, all of which are located internally of thehousing of said HMPD assembly. The lens can be other optical elementsthat may be used for imaging, including Fresnel lens, microlensletarrays, prisms, folding flat or curved mirrors, adaptive opticscomponents, micro-optics components, phase plates and any combinationsof the optical lenses. An additional preferred embodiment relates to amethod of forming a HMPD assembly comprising the steps of: positioningthe HMPD helmet on the user's head; displaying virtual images to saiduser's eye or eyes by a micro display disposed within said helmet;providing a phase conjugate material for also displaying said virtualimages from a display source integrated with the interior surface ofsaid helmet to said user's eye; and, providing an imaging lens, such asa Fresnel lens and others noted above for magnification of said phaseconjugate material whereby said magnified screen is projected to theuser's eye or eyes.

Further objects and advantages of this invention will be apparent fromthe following detailed description of presently preferred embodimentswhich are illustrated schematically in the accompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a concept illustrative cross-sectional view of the projectionhead mounted display (HMPD) assembly placed on the user's head, wherethe novel aspects of the invention are shown.

FIG. 2 shows the cross-sectional layout of the novel projection lenslayout of the invention.

FIG. 3 shows the residual ray aberrations in the image plane over pointsin the field of view.

FIG. 4 shows the longitudinal spherical aberration curves shifted on thelongitudinal axis denoting some residual lateral color occurring invisual space across the spectral wavelengths.

FIG. 5 illustrates the residual blur of the perceived image which showsto be about 1.3 arcmin at all points in the image vs. display location.

FIG. 6 shows the astigmatic field curves over the entire field of viewwith the residual lateral color.

FIG. 7 shows the residual lateral color smear vs. display location to beless than about 1 arcmin over the entire field of view.

FIG. 8 shows the astigmatism in arcminutes versus display location ofthe final image being projected from the miniature display inside theHMPD on the image plane.

FIG. 9 shows the amount of residual distortion to be about 1% over theentire field of view.

FIG. 10 shows the Diffraction MTF curves which illustrate how differentspatial frequency of a scene is perceived.

FIG. 11 shows the Scalar Diffraction Efficiency Estimate versus apparentheight of the diffraction optical element.

FIG. 12 shows the Scalar Diffraction Efficiency versus Wavelength of theDiffractive Optical Element (DOE) optical element.

FIG. 13 shows the surface profile of the DOE.

FIG. 14 shows how the HMPD can be attached to the user's head.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before explaining the disclosed embodiments of the present invention indetail, it is to be understood that the invention is not limited in itsapplication to the details of the particular arrangements shown sincethe invention is capable of other embodiments. Also, the terminologyused herein is for the purpose of description and not of limitation.

It would be useful to discuss the meanings of some words used herein andtheir applications before discussing the compact lens assembly of theinvention including:

-   HMPD—helmet mounted projection display;-   Singlet—single lens element;-   EFL—effective focal length;-   F^(#)—f-number;-   OAL—overall length;-   FOV—field of view (given in degrees for the diagonal of the    display);-   EPD—entrance pupil diameter;-   AMLCD—active matrix display;-   Conjugate—image of each other;-   Fresnel lens—a lens in which one collapses a surface into annular    zones to a thin plate;-   Microlenslet array—an array of miniature lenses comprised to replace    a conventional lens;-   Phase conjugate material—retro-reflective screen;-   Distortion—warping of the image;-   Arcminutes—an arcminute is the limit of visual acuity of the visual    human system with one degree visual angle corresponds to 60    arcminutes;-   Color Smear—a small spreading of the color spectrum in a point    image;-   Modulation—contrast;-   DOE—diffractive optical element; and,-   MTF—modulation transfer function.

Referring now to FIG. 1 of the instant Application, a miniature display501 is located beyond the focal point of a projection lens 502 which isused to display computer-generated images into a virtual environment.Rays traveling from the computer generated miniature active matrixdisplay 501 (exemplified by a 0.6 inch OLED microdisplay purchased fromeMagin Corporation) through the novel projection lens 502 (exemplifiedby an about 42 degree lens produced according to the disclosure of U.S.patent application Ser. No. 10/090,070 filed Mar. 1, 2002) provide anintermediary image 507 which is conjugate to the projected image 505.

When the phase conjugate material screen 504 (purchased from 3MCorporation) is at either the focal plane or within the focal plane ofthe lens 506 (commercially available from Edmund Scientific), itreflects rays at the same incoming angle in the reverse and oppositedirection traveling toward the beamsplitter 503 (commercially availablefrom Edmund Scientific) into the eye 509 of the user of the novel HMPDof the invention. When the lens 506 is placed at its focal plane andcombined phase conjugate material at optical infinity. For the case ofplacing the lens 506 within the focal plane, the phase conjugatematerial 504 is optically placed at a finite distance from the user'seye 509. The user's eye 509 will perceive the projected image 505 fromthe exit pupil 508 of the optical system. The unique novelty of the headmounted display of the invention is that all components, i.e., 501–506and 508–509, is within the helmet of the HMPD as indicated by the dottedlines of FIG. 1.

Refer now to FIG. 2 which shows in cross-section the projection lens 502referenced in FIG. 1. The lens 502 is composed of a two glass singletlenses, 510 and 514 respectively, two plastic singlet lenses, 511 and513 respectively, and the stop surface 512 which is in the middle ofglass-plastic and plastic-glass composition. In particular, the secondsurface of plastic singlet lens 511 is designed with a diffractiveoptical element (DOE), and the first surface of plastic singlet 513 isan aspherical surface. A single field flattener 515 is placed relativelyclose to the miniature display 501 to compensate field aberrations. Sucha novel optical design makes it possible to achieve compactness,light-weight (<10 g per eye), as well as good performance over thevisual spectrum.

As noted above with reference to lens 502, projective lens systems ofthis type are taught in co-pending U.S. patent application Ser. No.10/090,070 filed Mar. 1, 2002 of common assignee with the instantApplication and fully incorporated herein by reference thereto; and, inco-pending United States Patent Application designated UCF—380CIP-002filed Nov. 1, 2002 of common assignee also with the instant Applicationand fully incorporated herein by reference thereto.

The specification of the highly useful novel 42 degree lens 502 asdisclosed in the design system shown in FIG. 2 is:

Effective focal length (EFL)=19.5382 mm; F^(#)=1.62; Over-all-length(OAL)=25.6459 mm; Field of view (FOV)=42⁰; EPD=12 mm

The evaluation of the projective lens, shown in FIG. 2, has beenanalyzed and the resulting plots have been provided in FIGS. 3–4, 6,9–13 along with the visual performance graphs shown in FIGS. 5, 7 and 8.The overall assessment of the projective lens design is shown to havenegligible aberration in visual space.

FIG. 3 shows various points in the field 0, 0.3, 0.7 and 1 in order todetermine what residual aberrations are present in the referencedoptical lens system. The X-Y ray fan plot has a maximum vertical rangeof ±0.025 mm having residual aberrations which are further evaluated.

FIG. 4 quantifies spherical aberration across wavelengths. The shapes ofthe curves are the same for all three wavelengths meaning nospherochromatism. The curves are shifted on the longitudinal axisdenoting some residual lateral color will occur in visual space. Lateralcolor in visual space is further quantified in FIG. 9.

FIG. 5 shows the accommodation vs. display location the largest circlemeasures approximately 0.8 mm on the Figure, corresponding to about 1.3arcmin which is about human visual acuity.

FIG. 6 shows the astigmatic field curves of the projection lens whichare further evaluated in FIG. 8.

FIG. 7 shows lateral color smear vs. display location. The variation isabout 0.25 arcmin, which can not be resolved by the human eye; thereforeone can neglect lateral color.

FIG. 8 shows astigmatism curves expressed in arcminutes. The amount ofastigmatism results in about 1.2 arcminutes which is about human visualacuity.

FIG. 9 shows an amount of distortion of 1% at the edge of the field ofview. Distortion warps the virtual image displayed by either anelongation in the longitudinal image axis called barrel distortion orpincushion causing the sides of the virtual image to move inward.

FIG. 10 shows the MTF plot which has a design criterion of 20%modulation at 24 cycles/mm. In the design of the invention, it is shownthat at 24 cycles/mm, the modulation is above 60%. A minimum of 20% istypically required. Thus, this lens performance supersedes therequirements.

FIG. 11 shows the scalar diffraction efficiency which is estimated for alens radius of 5.517 mm at 98.7%.

FIG. 12 shows the scalar diffraction efficiency vs. wavelength for thenumber of zones “N” levels of the diffractive optical element (DOE),with a vertical axis as percentage and the horizontal axis representingthe visible spectrum.

FIG. 13 shows the continuous phase profile across the DOE radius in lensunit.

FIG. 14 shows a HMPD attached to a user's head and containing fully theintegration of the phase conjugate material 504, material and the lens,506 and as earlier emphasized in the discussion of FIG. 1 with respectto the 506 invention detailed herein, all the components, i.e., 501–506,and 508–509, which provides the virtual environment seen by the user'seye, 509 are located within the helmet (dotted lines) although one ormore of the components 501, 502, 503, 504, 506, 507, 508 and projectedimage, 505, can when appropriate be located outside of the helmet.

Refer again to FIG. 2 for showing of the final layout of the projectionlens. As therein,

-   501 —Miniature display-   502 —Projection lens-   510—Glass singlet 1-   511—Plastic singlet 1-   512—Stop-   513—Plastic singlet 2-   514—Glass singlet 2-   515—Single field flattener

TABLE 1 Optical lens specification Parameter Specification Object: ColorOLED a. Size Approximately 0.6″ inch in diagonal b. Active display areaRectangle, approximately 9 mm × approximately 12 mm c. Resolution 800 ×600 pixels Lens: a. Type Projection lens b. Effective focal lengthApproximately 19.5 mm c. Exit pupil diameter Approximately 12 mm d. Eyerelief Approximately 25 mm e. No. of diffractive surface Approximately 1Other Parameters: Wavelength range Approximately 656 to approximately486 nm FOV Approximately 42.0° in diagonal Distortion Approximately<2.0% over entire FOV

The nature of this invention is to incorporate projective optics andphase conjugate material without the inhibiting, hindering and limitingrequisite use of an external phase conjugate material to provide asee-through head mounted projection display. A key component of theinvention is not only the integration of the phase conjugate materialand projection optics within the HMPD but surprisingly also the use of alens in combination with this novel projection enclosed system markedlyfacilitates the operability of this revolutionary technology. Inprevious head mounted projection displays phase conjugate material hadto be placed in the environment to display images, but in this inventionthe user is not limited by the requisite use of an exterior phaseconjugate material.

Refer again to FIG. 1 for showing of the final layout of the componentswithin the HMPD which are identified with reference numbers 501 through507. As shown therein,

-   501—Miniature display-   502 —Projection lens-   503 —Beam splitter-   504—Phase conjugate material-   505—Projected image-   506 —Lens-   507 —Intermediary image-   508 —Entrance pupil-   509 —Eye

As shown in FIG. 1, the light from the miniature display 501 strikes thebeam splitter 503 after passing through the projection lens 502. Theminiature display 501 may display a virtual image as well as a computergenerated image. A portion of the light striking the beam splitter 503is reflected to produce the intermediary image 507. The remainder of thelight passes through the beam splitter 503 and lens 506 and produces theprojected image 505 (hollow arrow) on the phase conjugate material 504.The light from the projected image 505 is reflected back to the beamsplitter as shown by the upwardly directed arrows. The reflected lightstrikes the beam splitter 503 and is reflected toward the eye 509 withinthe area shown as the pupil entrance 508. The lens 506 can be an opticalelement such as a Fresnel lens, microlenslet array, prism, flat mirror,curved mirror, folding mirror, phase plate, adaptive optic component,micro-optics component and micro-phase plate component or anycombination of the optical lenses. Placement of the lens 504 and phaseconjugate material 504 at a location outside of the user's line of sightextends usage to see-through augmented reality to produce images usingthe see-through head mounted projective display system.

The invention improves upon not being limited to use of the phaseconjugate material in the environment but dramatically extends the useof outdoor see-through augmented reality. Furthermore, this inventionextends the use of projection head mounted displays to clinical guidedsurgery, medical surgery, outdoor augmented see-through virtualenvironment for military training and wearable computers, and for usewith binoculars. In these latter applications, a head mounted projectiondisplay (HMPD) optical lens assembly comprising in combination aprojection lens having a field of view (FOV) of up to approximatelyninety degrees and of an overall weight of less than approximately 10grams; and a micro display ranging from approximately 0.2 inches up toapproximately 2 inch diagonal size whereby an intermediate image will beviewed by the user's eye is surprisingly and particularly useful.

While the invention has been described, disclosed, illustrated and shownin various terms of certain embodiments or modifications which it haspresumed in practice, the scope of the invention is not intended to be,nor should it be deemed to be, limited thereby and such othermodifications or embodiments as may be suggested by the teachings hereinare particularly reserved especially as they fall within the breadth andscope of the claims here appended.

1. A head mounted projection display (HMPD) optical compact lensassembly for monocular use comprising: (a) a display for displaying acomputer generated image; (b) a projection lens for projecting virtualimages and said computer generated image to a user's eye in a monocularsetting; (c) only one phase conjugate component for receiving andprojecting said virtual images; and (d) an optical lens for imaging ofsaid phase conjugate component, wherein the display, projection lens,phase conjugate component and the lens are within the HMPD opticalcompact lens assembly.
 2. The HMPD assembly of claim 1 wherein saidoptical lens for imaging is a Fresnel lens.
 3. The HMPD assembly ofclaim 1 wherein said optical lens for imaging is a microlenslet array.4. The HMPD assembly of claim 1 wherein said optical lens for imaging isa prism.
 5. The HMPD assembly of claim 1 wherein said optical lens forimaging is a flat mirror.
 6. The HMPD assembly of claim 1 wherein saidoptical lens for imaging is a curved mirror.
 7. The HMPD assembly ofclaim 1 wherein said optical lens for imaging is a phase plate.
 8. TheHMPD assembly of claim 1 wherein said optical lens for imaging is anadaptive optics component.
 9. The HMPD assembly of claim 1 wherein saidoptical lens for imaging is a micro-optics component.
 10. The HMPDassembly of claim 1 wherein said optical lens for imaging is amicro-phase plate.
 11. The HMPD assembly of claim 1 wherein said opticallens is selected from at least one of: Fresnel lens; microlenslet array;prism; flat mirror; curved mirror; phase plate; adaptive opticscomponent; micro-optics components; and, micro-phase plate.
 12. The HMPDassembly of claim 1, wherein said assembly includes a folding of theoptical means.
 13. The HMPD assembly of claim 1 wherein said display isapproximately 0.6 inch diagonal.
 14. A head mounted projection display(HMPD) optical compact lens assembly for binocular use comprising: (a) ameans for displaying a computer generated image onto a micro display;(b) optical means for projecting virtual images and said computergenerated display to a user's eye in a binocular setting to provide asee-through augmented reality; (c) only one phase conjugate materialcomponent within the HPMD assembly for receiving and projecting saidvirtual images; and (d) optical means for imaging of said phaseconjugate material component wherein the displaying means, opticalprojection means, phase conjugate component and the optical imagingmeans are within the HPMD assembly.
 15. The HMPD assembly of claim 14wherein said optical means for imaging is a Fresnel lens.
 16. The HMPDassembly of claim 14 wherein said optical means for imaging is amicrolenslet array.
 17. The HMPD assembly of claim 14 wherein saidoptical means for imaging is a prism.
 18. The HMPD assembly of claim 14wherein said optical means for imaging is a flat mirror.
 19. The HMPDassembly of claim 14 wherein said optical means for imaging is a curvedmirror.
 20. The HMPD assembly of claim 14 wherein said optical means forimaging is a phase plate.
 21. The HMPD assembly of claim 14 wherein saidoptical means for imaging is an adaptive optics component.
 22. The HMPDassembly of claim 14 wherein said optical means for imaging is amicro-optics component.
 23. The HMPD assembly of claim 14 wherein saidoptical means for imaging is a micro-phase plate.
 24. The HMPD assemblyof claim 14 wherein said optical means is selected from at least one of:Fresnel lens; microlenslet array; prism; flat mirror; curved mirror;phase plate; adaptive optics component; micro-optics component; and,micro-phase plate.
 25. The HMPD assembly of claim 14 wherein saidassembly includes a folding of the optical means.
 26. The HMPD assemblyof claim 14 wherein said micro display is approximately 0.6 inchdiagonal.
 27. A method of viewing internally generated virtual images ina head mounted projection display (HMPD) using an optical lens assemblycomprising the steps of: (a) positioning a HMPD helmet on a user's head;(b) displaying virtual images to a user's eye by optical display meansdisposed within said helmet; (c) providing only one phase conjugationmaterial component below the eyes of the user for reflecting saidvirtual images from a micro display source integrated with an interiorsurface of said helmet to said user's eye; and (d) providing a lensmeans for magnification of said phase conjugate material componentwhereby said magnified screen means is further projected to the user'seyes.
 28. A method of viewing internally generated virtual images in ahead mounted projection display (HMPD) using an optical lens assemblycomprising the steps of: (a) positioning a HMPD helmet on a head of auser; (b) displaying virtual images to an eye of the user by opticaldisplay means disposed within said helmet; (c) incorporating only onephase conjugate component for receiving and projecting said virtualimages from a miniature display inside the helmet; and (d) incorporatinga lens inside the helmet for imaging of said projected virtual imagesfrom said phase conjugate component, wherein said virtual image arefurther projected to said eye of said user.
 29. A head mountedprojection display (HMPD) assembly comprising: (a) a miniature displayfor displaying an image; (b) a projection lens for projecting said imagefrom said miniature display; (c) only one phase conjugate componentbelow said projection lens for receiving and reflecting said projectedimage from said projection lens; and (d) an imaging lens within thefocal plane of said phase conjugate component for magnifying of saidphase conjugate component, wherein a reflected virtual image isprojected to an eye of a user.